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Overview 



What are the key factors that promote academic success among students whose demo- 
graphic characteristics and school circumstances place them at high risk of failure? This paper 
provides highly suggestive, although not conclusive, answers to this question. Through path 
analysis modeling techniques applied to data collected in MDRC’s evaluation of the First 
Things First school reform initiative in a large urban school district, the paper explores the in- 
fluence of two psychological variables — student engagement and perceived academic compe- 
tence — on achievement in reading and mathematics. 

This study’s findings may have important implications for understanding how students 
learn in the classroom. Consonant with previous research, they indicate that both engagement in 
school and students’ perception of their own academic competence influence achievement in 
mathematics for high school students. But the study departs from earlier work in suggesting that 
perceived academic competence may be more influential than engagement in boosting achieve- 
ment in both mathematics and reading. Indeed, analyses indicate that perceived competence had a 
stronger influence on subsequent engagement than engagement had on students’ perceptions of 
themselves as competent learners. 

The findings also make clear that supportive teachers and clear and high expectations 
about behavior are key to the development of both student engagement and perceived compe- 
tence. This study suggests that the earlier schools and teachers begin to build students’ confi- 
dence in their ability to do well, the better off students will be. Because students’ perceptions of 
their capacity for success are key to their engagement in school and learning, schools should be 
designed to enhance students’ feelings of accomplishment. Teachers whom students see as sup- 
portive and who set clear expectations about behavior help create an atmosphere in which stu- 
dents feel in control and confident about their ability to succeed in future educational endeavors. 
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Introduction 

Much research in recent years has focused on identifying the key factors that promote 
academic success among students whose demographic characteristics and school circumstances 
place them at high risk of failure. In large part, this research has addressed the characteristics of 
individual students and school settings that are optimal for success. Literature largely supports the 
positive role that students’ attitudes and behavior play in improved academic achievement. Sev- 
eral studies have found that engagement in school and perceived academic competence (that is, 
positive feelings about one’s ability to be successful academically) strongly predict improved 
reading and mathematics achievement. Similarly, literature supports the positive influence of fac- 
tors in the school context — for example, the presence of high-quality, engaging instructional ac- 
tivities and supportive adult relationships — in improving students’ academic outcomes. 

This paper examines the relationships among these three constructs — school context, 
student attitudes and behavior, and achievement — using longitudinal data from a large-scale 
high school reform effort. The analysis is exploratory in nature, in that it tests one particular hy- 
pothesis about the relationships among these constructs. Other hypotheses may be equally plau- 
sible, but this paper considers the relationships shown in Figure 1, which presents the theoretical 
model underlying the effort. 

Student attitudes and behavior stand at the center of the figure and the theory that un- 
derlies it. As the figure indicates, it is hypothesized that student attitudes and behavior (1) con- 
tribute to mathematics and reading achievement among high school students, and (2) result 
from key factors in the school context: support from teachers; clear, high, and consistent expec- 
tations; and high-quality instruction. That is, the positive influence of school context on im- 
proved achievement is mediated by students’ attitudes about themselves as learners and by be- 
havior that is correlated with academic success. 

The figure suggests two major research questions that frame the analyses in later sections: 

1 . What is the influence of the two psychological variables — engagement in 
school and perceived academic competence — on student achievement in 
reading and mathematics? 

2. If these psychological variables do have a positive influence on achieve- 
ment, which elements of the school context support the development of 
higher levels of student engagement and perceived academic competence? 
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School Context, Student Attitudes and Behavior, and Academic Achievement 

Figure 1 

Theoretical Model 




Student Background and Previous Achievement 



In addition, the research addresses two secondary questions: 

3. What is the directionality of the relationship between perceived academic 
competence and student engagement — that is, does perceived academic 
competence influence engagement or vice versa? 

4. What is the directionality of the relationships between variables in school 
context and the psychological variables? 

In Figure 1, the variables of paramount interest to the study are enclosed in solid-line 
boxes. As the figure indicates, students' background characteristics and their levels of prior 
achievement influence their subsequent achievement, attitudes and behavior, and perceptions of 
school context. While the analysis takes these factors into account, it does not focus on them; in 
the figures shown in this paper, dotted-line boxes surround these two sets of variables. 

The next section offers a brief overview of the relevant literature on student attitudes 
and behavior and their relationship to academic achievement and on elements of school context 
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that are associated with student success. This is followed by an overview of the research design: 
the sample of students followed in the study, the measures and their sources, and the analytical 
model. The study’ s findings are presented, and the paper concludes by discussing the practical 
implications of these findings for educators and policymakers. 



Contextual and Psychological Predictors of Student Learning and 
Success: A Review of the Literature 

There is substantial evidence that engagement in school is important in promoting stu- 
dent success and learning and that a number of factors in the school environment foster high 
levels of engagement. Research also suggests that the influence of the educational context on 
engagement is partially mediated by psychological beliefs about competence and control. The 
next sections briefly summarize the literature relevant to these points. 

Engagement and Learning 

Student engagement can be defined as the level of participation and intrinsic interest 
that a student shows in school . 1 Engagement in schoolwork involves both behaviors (such as 
persistence, effort, attention) and attitudes (such as motivation, positive learning values, enthu- 
siasm, interest, pride in success ). 2 Thus, engaged students seek out activities, inside and outside 
the classroom, that lead to success or learning. They also display curiosity, a desire to know 
more, and positive emotional responses to learning and school . 3 

Extensive evidence exists that engagement and motivation are critical elements in student 
success and learning. Researchers agree that engaged students learn more, retain more, and enjoy 
learning activities more than students who are not engaged . 4 Studies have shown a direct link be- 
tween levels of engagement and achievement in reading and mathematics . 5 Many school-level 
studies have identified higher levels of student engagement as important predictors of scores on 
standardized achievement tests, classroom learning and grades, and student persistence . 6 



'Newmann ( 1992). 

2 Connell and Wellborn (1991); Johnson, Crosnoe, and Elder (2001); Newmann (1992); Skinner and Bel- 
mont (1993); Smerdon (1999); Turner, Thorpe, and Meyer (1998). 

3 Newmann (1992). 

4 Dowson and Mclnemey (2001); Hancock and Betts (2002); Lumsden (1994). 

5 Kirsch et al. (2002). 

6 National Research Council (2000). As important as student engagement has been found to be in support- 
ing and leading to learning, a substantial number of high school students, particularly those in urban school 
systems with large numbers of poor, minority students, are disengaged from school. This jeopardizes their abil- 
ity to learn and advance through the educational system. Some studies have found that 40 to 60 percent of high 
school students are chronically disengaged, as exhibited by inattentiveness, lack of effort, inability or unwill- 

(continued) 
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Perceived Competence and Control as Mediators of the Relationship 
Between School Context and Student Engagement 

A primary psychological mediator of the relationship between student engagement and 
educational context is the degree to which students feel competent and confident of their ability 
to be successful in completing educational tasks. Students who are convinced that they lack the 
ability to succeed or control the outcome of their educational experience will not make an effort 
to engage or excel in school-related work . 7 To become successful, students need to know what 
it takes to succeed and believe they can succeed, given what they know. Thus, a student who 
doesn’t think she can complete assigned homework successfully, or who doesn’t understand 
what to do, is unlikely to attempt the assignment. Similarly, a student who doesn’t think he will 
be able to pass the courses needed to graduate is unlikely to do much work and may end up cut- 
ting class or even dropping out. 

Students' beliefs about their competence and their expectations for success in school 
have been directly linked to their levels of engagement, as well as to emotional states that pro- 
mote or interfere with their ability to be academically successful. For example, students who 
believe they are academically incompetent tend to be more anxious in the classroom and more 
fearful of revealing their ignorance . 8 They fear that educational interactions will result in embar- 
rassment and humiliation, and this, in turn, inhibits them from behaving in ways that might help 
them, such as asking questions when they are confused or engaging in trial-and-error problem- 
solving . 9 In addition, such students are more likely to avoid putting much effort into a task so 
that they can offer a plausible alternative to low ability or lack of knowledge as an explanation 
for failure — for example, “I could have done it if I tried, but I didn’t feel like doing it .” 10 

Factors in the School Context that Support Student Success 

Research suggests that variables in the educational context are important in supporting 
and sustaining positive academic self-perceptions and engagement in school . 11 This research 
base is mostly qualitative, correlational, or quasi-experimental and falls short of the random as- 
signment design that some researchers believe is necessary to draw causal conclusions. Never- 



ingness to compete educational tasks and assignments, and self-reported levels of boredom. This figure takes 
into account only students who are still in school, not those who have dropped out (Marks, 2000; Sedlak, 
Wheeler, Pu 1 1 in, and Cusick, 1986; Steinberg, 1996). The proportion of low-income, minority, urban students 
who report being disengaged is even higher (National Research Council, 2003). 

7 Atkinson (1964); Eccles et al. (1983); Skinner, Wellborn, and Connell (1990); Skinner, Zimmer- 
Gembeck, and Connell (1998). 

8 Abu-Hilal (2000); Bandalos, Yates, and Thorndike-Christ (1995); Harter (1992); Hembree (1988). 

9 Newman and Goldin ( 1990); Ryan and Pintrich (1997). 

10 Covington, Spratt, and Omelich ( 1980). 

1 'National Research Council (2003). 
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theless, the evidence is consistent enough to suggest that school context plays an important role 
in student learning and achievement through its relationship with student engagement. 

Relationships between students and teachers and the climate in the classroom are posi- 
tively associated with levels of student engagement and academic competence. Similarly, mean- 
ingful and challenging learning environments have been linked to both engagement and per- 
ceived competence. When students are authentically engaged in meaningful, quality work, the 
likelihood increases that they will learn something new and remember what they learned . 12 

Three kinds of contextual factors merit special attention: 

Sense of Belonging and Caring 

Although learning involves individual cognitive and emotional processes, student motiva- 
tion is also significantly influenced by a supportive network of relationships. The likelihood that 
students will be motivated and engaged in school is increased to the extent that they perceive their 
teachers, family, and friends as supportive. Schools that engage students promote a sense of be- 
longing by personalizing instruction and creating a supportive, caring social environment where 
adults show an interest in students’ lives in and out of school . 13 The research on belonging in edu- 
cational contexts is relatively new, and the direction of causality has not been definitively estab- 
lished . 14 Nevertheless, many correlational and nonexperimental studies have shown that students 
who report caring and supportive interpersonal relationships in school have more positive aca- 
demic attitudes and values and are more satisfied with school . 15 Such students also are more likely 
to attend school, learn more , 16 and report that they are more engaged in academic work . 17 

Clear, High, and Consistent Expectations 

High, clear, and consistent expectations also support students’ self-confidence, their be- 
lief that their efforts will lead to success, and their engagement in school . 18 A substantial body 
of evidence demonstrates that schools where students achieve high levels of performance tend 
to set high expectations and standards . 19 To motivate students, however, standards and expecta- 



12 Hancock and Betts (2002); Willm s (2002). 

13 National Research Council (2003). 

14 National Research Council (2003). 

15 Baker (1999); Battistich, Solomon, Kim, Watson, and Schaps (1995); Ryan and Deci (2000); Shouse 
(1996); Skinner and Belmont (1993); Wasley et al. (2000); Yowell ( 1999). 

16 Bryk and Driscoll ( 1988); Bryk, Lee, and Holland ( 1993). 

17 Connell and Wellborn (1991). 

18 Eccles et al. (1983); Gambone, Klem, Summers, and Akey (2004); Wigfield and Harold ( 1992). 

19 Baker, Terry, Bridger, and Winsor (1997); Evans (1997); Lambert and McCombs (1998); Lee, Bryk, 
and Smith ( 1993); Lee and Smith (1999); Phillips (1997). 
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tions must be clear and genuinely achievable. 20 Students are most likely to be academically en- 
gaged when goals are set at an appropriate level — that is, when they both challenge students 
and allow them to experience a sense of competence and accomplishment. 

Meaningful and Challenging Educational Environments 

Research on learning shows that students become cognitively engaged when teachers ask 
them to wrestle with new concepts, explain their reasoning, defend their conclusions, or explore 
alternative strategies and solutions. 21 Students enjoy learning more and are more likely to partici- 
pate in school tasks when their teachers employ active pedagogical strategies. Collaboration 
among peers — students working together in pairs or small groups to help one another learn — 
also has been associated with increased engagement and learning. 22 When students can put their 
heads together rather than work in isolation, they are more receptive to challenging assignments. 23 

Research indicates that over the long term, students are more likely to be engaged in the 
classroom when they are asked to conduct experiments, participate in debates and role-playing, 
create models, and complete projects. 24 Evidence also suggests that when classroom instruction 
draws on students’ preexisting knowledge, culture, and real-world experiences, it becomes more 
meaningful. 25 Students enjoy learning more and learn better when what they are studying is of 
personal interest and relates to their lives. 26 



Research Methods and Design 

This section describes the study’ s design. It discusses the sample and data and the ana- 
lytic methods and models used to address the research questions. 

The Research Sample 

This analysis draws on student surveys and administrative records data collected in 
MDRC’s evaluation of the First Things First school reform initiative in a large urban school 
district during the 2001-2002, 2002-2003, and 2003-2004 school years. 27 Students in the analy- 



20 Gambone et al. (2004); Lee and Smith (1999); Phillips ( 1997). 

21 National Research Council ( 1999). 

22 Davidson (1999); Johnson and Johnson (1985); Mitchell (1993). 

23 Cohen (1994). 

24 Davidson (1999); Guthrie and Wigfield (2000); Mitchell (1993). 

25 McLaughlin and Talbert (1993). 

26 Meece (1991). 

27 First Things First is a schoolwide reform initiative that has been implemented in multiple sites across the 
United States, including Kansas City (Kansas), Houston, St. Louis, and rural Mississippi. The initiative is 
based on structural changes in staffing and teaming; small learning communities; changes in leadership and 

(continued) 
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sis sample attended three high schools in this district. Two of the schools had implemented First 
Things First for two years and one for three years when data collection ended; at all three 
schools, a year-long planning period preceded program implementation. 

The sample consisted of 449 students for whom academic achievement measures were 
available for all three years of the study. Survey data for all students were available only for the 
second and third years. Because the sample includes only students who remained in school 
throughout the study period, sample members are not fully representative of all high school stu- 
dents in the district, which is marked by high dropout rates. 

Sixty-four percent of the students were tenth-graders and 36 percent were eleventh- 
graders in the last year of the study. (Seniors were excluded from the analysis because the dis- 
trict did not administer reading and mathematics tests to twelfth-graders.) Males comprised half 
the sample. Sixty-seven percent of the students were Hispanic, 24 percent African-American, 4 
percent white, and 5 percent other. Approximately 85 percent of the students received free or 
reduced-price lunch (an indicator of low socioeconomic status), 10 percent participated in spe- 
cial education programs, and 23 percent were in English for Speakers of Other Languages 
(ESOL) programs. 

Measures and Data Sources 

As summarized in Table 1, the analysis used data from surveys and administrative re- 
cords to examine four broad sets of variables. Three of these categories of variables — aca- 
demic achievement, student attitudes and behavior, and school context — correspond to the key 
constructs of interest in Figure 1 . In the fourth category are student demographic characteristics, 
which were also taken into account in the analyses. 

Administrative records contain test score data that are used in two ways: Year 1 test 
scores offer measures of prior reading and mathematics achievement, while Year 3 test scores pro- 
vide information on subsequent achievement in these subjects. Administrative records also yield 
data on student background characteristics, as measured during Year 3. (Two of the five back- 
ground characteristics measured — gender and race/ethnicity — represent invariant characteris- 
tics; low socioeconomic status, as measured by receipt of free- or reduced-price lunch, is also 
likely to have remained stable over time.) Survey data from both Year 2 and Year 3 provided 
measures of students' perceptions of the school context and their degree of engagement in school 
and perceived academic competence. These data are described in greater detail in Table 1. 



professional development; aligned, standards-based curriculum and assessment; and high-quality instructional 
approaches. For more information on the First Things First initiative, see 
http://www.mdrc.org/pubhcations/412/overview.html. 
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School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 1 



Constructs Measured, Data Sources, and Timeline for Data Collection 



Construct Measured 


Doto VAIll^PO _ 


Years Measured 


Ddld ijUlirLc 


1 2 


3 


Student achievement 


Administrative records 






Prior achievement 




X 




Subsequent achievement 






X 


Student attitudes and behavior 


Student surveys 






Engagement 




X 


X 


Perceived academic competence 




X 


X 


School context 


Student surveys 






Teacher support 




X 


X 


Clear, high, and consistent academic 








and behavioral expectations 




X 


X 


High-quality pedagogy 




X 


X 


Student demographic characteristics 


Administrative records 






Gender 






X 


Race/Ethnicity 






X 


Free or reduced-price lunch 






X 


School 1 






X 


Grade level 






X 



NOTES: ’in no analysis did the school attended emerge as a statistically significant predictor; therefore it has 
been dropped from the subsequent tables. 



Student Achievement 

Scores on the reading and mathematics subtests of the Stanford Achievement Test 
(SAT) served as measures of student achievement. The SAT is a standardized achievement test 
administered to all students in the district in grades 1 to 1 1 . The district administered one ver- 
sion of the test, the SAT-9, during the first two years covered by the study, changing to the 
SAT-10 in the third year; Year 3 scores were transformed into SAT-9 norms. Test scores are 
expressed in terms of normal curve equivalents (NCEs). 28 



28 Normal Curve Equivalent (NCE) scores have many of the same characteristics as percentile ranks, but 
are based on an equal-interval scale — that is, the difference between two successive scores has the same 
meaning at any two points along the scale. 







Year 1 scores indicate students’ achievement in grade 8 or grade 9. Year 3 scores indi- 
cate students’ achievement in grade 10 or grade 11. 

Student Attitudes and Behavior 

Student survey responses were used to create two scales of student attitudes and behav- 
iors: the Student Engagement Scale and the Perceived Academic Competence Scale. The Ap- 
pendix shows the items that make up these scales, as well as those pertaining to school context, 
as discussed below. 

The Student Engagement Scale is an index of how hard students work in school and 
their level of participation in activities associated with academic success. It consists of five 
items, including, “I work very hard on my homework” and “I don’t try very hard at school.” 
The Student-Perceived Academic Competence Scale is an index of students’ perceptions about 
how successful they can be in school and the degree of control they have over their academic 
success. It contains nine items, including, “I can do well in school if I want to,” “Trying hard is 
the best way for me to do well in school,” and “I’m pretty smart in school.” Internal consistency 
reliabilities for the indices ranged from .7 1 to .74 for School Engagement and .79 to .78 for Per- 
ceived Academic Competence in Years 2 and 3, respectively. 

School Context 

The student surveys also yielded measures of students' perceptions of three important 
aspects of the school environment: teacher support; clear, high, and consistent expectations; and 
high-quality pedagogy. 

The Teacher Support Scale is an index of how much students feel that their teachers 
support them and like them. It consists of eight items, including, “My teachers interrupt me 
when I have something to say” and “My teachers care about how I do in school.” Internal con- 
sistency reliabilities for the index ranged from .77 to .78 in Years 2 and 3, respectively. 

Two indices assess the presence of clear, high, and consistent expectations. Academic 
expectations define what all students should know and be able to do within and across key con- 
tent areas when they leave high school and at points along the way in their school careers. The 
Academic Expectations Scale contains 14 items, including, “Your teacher makes clear to you 
that you are expected to come to class prepared,” “Everybody is expected to take part in class- 
room activities,” and “Your teacher makes clear to you examples of high-quality work that will 
lead to high grades.” 

Conduct expectations define how adults and students should behave; there are clear 
benefits for meeting these standards and consequences for violating them. The Conduct Expec- 
tations Scale consists of 11 items, including, “Your teacher makes clear to you what student 



9 




conduct is unacceptable,” “The rules in this school are very clear,” and “Students in my school 
are expected to treat all of the adults in this school with respect all of the time.” Internal consis- 
tency reliabilities ranged from .80 to .85 for Academic Expectations and .68 to .72 for Conduct 
Expectations in Years 2 and 3, respectively. 

High-quality pedagogy is assessed by three indices of instructional practice: active 
learning strategies, making connections and extensions, and student-to-student interactions. Ac- 
tive learning is defined as engaging students in pairs or small groups to write, discuss, and/or 
manipulate learning equipment or materials. The Active Learning Strategies Scale contains 1 1 
items. For example, students were asked, with respect to their English or math class, “How of- 
ten do students in your class discuss and ask each other questions about the work you're do- 
ing?” and “How often do students in your class work on projects that last several class periods?” 

Making connections and extensions is defined as the degree to which students partici- 
pate in activities that require them to connect what they are learning to real-world circum- 
stances, solve novel problems, and extend their knowledge to different situations. The Making 
Connections and Extensions Scale consists of four items, including, “When you are learning 
English for math], how much do your teachers emphasize learning to find more than one way of 
approaching a problem or issue?” and “How much do your teachers emphasize making connec- 
tions between what goes on inside and outside of school?” 

Student-to-student interactions are defined as the level of interactions between students 
in classroom activities that are designed to promote learning. The Student-to-Student Interac- 
tions Scale has five items, including, “In your English [or math] class, how true is it that stu- 
dents go over and discuss each other’s work?” and “How true is it that students review what 
they’ve learned with one another?” Internal consistency reliabilities ranged from .71 to .75 for 
Active Learning, .68 to .7 1 for Making Connections, and .76 to .74 for Student-to-Student Inter- 
actions in Years 2 and 3, respectively. 

Data Analysis Strategy 

The study employs path analysis modeling techniques. Path analysis is an extension of 
regression analysis, a statistical technique for producing a quantitative estimate of how much 
one variable (the “independent variable”) influences another variable (the “dependent vari- 
able”). Path analysis models specify hypothesized associations or directional relationships 
among a set of sequentially ordered variables. A given model is usually depicted as a set of cir- 
cles or boxes representing the variables that are connected by one-way arrows, which indicate 
the hypothesized direction of causation. Each variable in a path analysis model is considered as 
the dependent variable in a regression analysis in which antecedent variables are treated as in- 
dependent variables; at the same time, a variable that serves as a dependent variable in one part 



10 




of the model may serve as an independent variable in another part of the model. The strength of 
path analysis is that particular influences can be examined while controlling for all other vari- 
ables in the model; both direct and indirect influences can be examined. 29 

Path analysis is used to test various causal models that the researcher is comparing. The 
regression weights predicted by each analysis model are compared with the correlation matrix 
for the variables, and a goodness-of-fit statistic is calculated. The best-fitting of two or more 
models is selected by the researcher as the best model for advancement of the theory. 

Path analysis requires the usual assumptions of regression analysis. It is particularly 
sensitive to model specification, because failure to include relevant causal variables or the inclu- 
sion of extraneous variables often substantially affects the path coefficients, which are used to 
assess the relative importance of various direct and indirect causal paths to the dependent vari- 
able. Such interpretations should be undertaken in the context of comparing alternative models, 
after assessing their goodness of fit. 

By using EQS, a latent variable modeling regression software program, 30 analyses 
were conducted that took into account the residual error of each predictor and criterion vari- 
able in the model. This accounting for residual error allowed the relationships among latent 
constructs to be modeled by removing sources of error that may have had an adverse influ- 
ence on the statistical findings. 

Different path analysis models address the four research questions set out at the begin- 
ning of the paper. 

Figure 2 depicts the model used to answer the first question: 

• What is the influence of the two psychological variables — engagement in 
school and perceived academic competence — on student achievement in 
reading and mathematics? 

This analysis estimated the longitudinal influence of student engagement and perceived 
academic competence on achievement. Year 3 mathematics and reading achievement scores 
were the critical outcomes of interest (the dependent variable), while engagement and perceived 
academic competence, measured in Year 2, were the key predictor (independent) variables. 
Separate analyses were conducted for reading and mathematics achievement. 



29 The direct effect is the partial coefficient (beta for standardized, b for unstandardized) for y on x control- 
ling for all prior variables and all intervening variables in the model. The indirect effect is the total causal effect 
minus the direct influence, and measures the influence of the intervening variables. 

30 EQS-Stmctural Equation Modeling Software, Multivariate Software Inc. 
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School Context, Student Attitudes and Behavior, and Academic Achievement 



Figure 2 

Influence of Student Attitudes and Behavior 
on Student Academic Achievement 



Prior 

achievement 
Year 1 




Like the others analyses described below, this analysis controlled for prior academic 
achievement (measured in Year 1) and student demographic and other characteristics (measured in 
Year 3) — variables that would be expected to be highly associated with the dependent variable. 31 

Other path analysis models tackled the second research question: 

• If these psychological variables do have a positive influence on achievement, 
which elements of the school context support the development of higher lev- 
els of student engagement and competence? 

Here, the key outcomes of interest are school engagement and perceived academic 
competence, as measured by the Year 3 surveys. Because the influence of school context on 
engagement in school and perceived academic competence may be either long term or immedi- 
ate, two different analyses were conducted. The lagged analysis, shown in Figure 3a, examined 
the relationships between students’ perceptions of the school context in Year 2 and their psy- 
chological dispositions in Year 3. In contrast, in the concurrent model, shown in Figure 3b, stu- 
dents’ perceptions of school context and their psychological dispositions were measured in the 
same year, Year 3. 

'The measure of prior achievement is the average of reading and math NCE scores for Year 1. 
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School Context, Student Attitudes and Behavior, and Academic Achievement 



Figure 3a 

Lagged Model of Influence of School Context on Student Engagement and 
Perceived Academic Competence 




L 



Student 
demographics 
Year 3 



School Context, Student Attitudes and Behavior, and Academic Achievement 

Figure 3b 

Concurrent Model of Influence of School Context on Student 
Engagement and Perceived Academic Competence 




Student 
demographics 
Year 3 
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Yet another path analysis model was used to address the third research question: 

• What is the directionality of the relationship between perceived academic 
competence and student engagement? 

Did students who were initially more engaged in their schoolwork come to see them- 
selves as more academically competent? Or did students who initially viewed themselves as 
more academically competent come to feel more engaged? Figure 4 shows this cross-lagged 
analysis. 32 Cross-lagged models allow researchers to simultaneously test the directional influ- 
ence of two variables on one another over time and to make judgments about the relative 
strength of the influences. 

A final set of models addressed the fourth question the study seeks to answer: 

• What is the directionality of the relationships between school context vari- 
ables and the psychological variables? 



School Context, Student Attitudes and Behavior, and Academic Achievement 

Figure 4 

Cross-Lagged Model of Relationship Between Student Engagement and 
Perceived Academic Competence 



1 

i- 

Prior i 

achievement | 

Year 1 J 
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j Student 

i demographics 
i Year 3 

I 

i 



32 The model in Figure 4 also controlled for school context variables in the analysis. 
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Did students who felt more supported by their teachers in Year 2 display greater en- 
gagement in Year 3? Or, conversely, did students who were more engaged in Year 2 perceive 
their teachers to be more supportive in Year 3? This analysis entailed another set of models in 
which the school context factors that were found to be statistically significant predictors of the 
psychological variables were cross-lagged with student engagement and with perceived aca- 
demic competence. These models follow the general structure shown in Figure 5. 



School Context, Student Attitudes and Behavior, and Academic Achievement 

Figure 5 

Cross-Lagged Model of Relationship Between Student Engagement and 

Teacher Support 



Prior 

achievement 
Year 1 




Student 
demographics 
Year 3 



Different schools may offer their students different learning experiences, and these, in 
turn, may contribute to differences in student achievement, engagement, and perceived aca- 
demic competence. To control for these potential differences on the dependent variable, a series 
of fixed school influences were entered into the model. These school-level fixed influences 
were included in all analyses. 

It is important to acknowledge at the outset that while the purpose of the path analysis 
models is to provide estimates of causal relationships among variables, path analysis cannot 
establish definitively that one variable indeed has a causal role with respect to another variable. 



15 








At best, it can provide evidence that is consistent with causal explanations. In the text below, the 
term “influence” is used to describe the putatively causal role of one variable with respect to 
another — that is, Variable A is said to “influence” Variable B when evidence is consistent with 
the notion that A caused B . 



Results 

This section presents the study’s findings. It begins by presenting descriptive statistics 
on the variables used in the analyses. Attention then turns to the four questions that underlie the 
study. The implications of the findings are discussed in the final section of the paper. 

All of the analyses below control for student demographic characteristics and prior 
achievement in examining the relationships between the key variables of interest. To avoid re- 
dundancy, only additional control variables introduced in the analyses are noted in the text. 

Descriptive Statistics 

Table 2 shows the mean and standard deviation of each variable included in the analy- 
ses described above for each year in which the variable was measured. As noted above, the stu- 
dent achievement measures are NCE scores on the SAT-9 and SAT-10, while the measures of 
student attitudes and behavior and school context are scales whose value ranges from 1 to 4 
(with higher values representing a greater quantity of a particular construct). 

Research Question 1 : The Influence of School Engagement and Perceived 
Academic Competence on Academic Achievement 

The first research question examined the influence of student engagement and per- 
ceived academic competence on reading and math achievement. The key findings follow: 

• Finding 1. Both prior student engagement and perceived academic compe- 
tence had a significant positive influence on subsequent levels of math 
achievement, but the influence of perceived academic competence was three 
times larger than that of engagement. 

• Finding 2. Perceived academic competence had a positive influence on read- 
ing achievement; the influence of engagement was more complex. 

• Finding 3. Prior achievement was also significantly related to perceived 
competence, suggesting that students who do well on reading and mathemat- 
ics assessment tests then perceive themselves as able learners, which pro- 
motes more reading and mathematics success. 
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Tables 3 and 4 provide the detailed results of the analyses for math and reading outcomes, 
respectively. The tables show the standardized regression coefficient of each independent variable. 
The standardized regression coefficient can be interpreted as the standard deviation change in the 
dependent variable associated with one standard deviation change in the independent variable. 
The tables also show the level of statistical significance associated with each coefficient. 



School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 2 

Means and Standard Deviations of All Analysis Variables 



A n olvcic Voi'idhioc _ 


Year l a 


Year 2 


Year 3 


illldljMa VdlldUlca 


Mean 


SD 


Mean 


SD 


Mean 


SD 


Student achievement 


SAT-9/10 mathematics achievement 


40.23 


15.48 


NA 


NA 


43.66 


13.16 


SAT-9/10 reading achievement 


34.31 


17.64 


NA 


NA 


38.93 


17.91 


Student attitudes and behavior 


Engagement 


NA 


NA 


3.19 


0.49 


3.26 


0.44 


Perceived academic competence 


NA 


NA 


3.2 


0.54 


3.27 


0.57 


School context 


Supportive teacher relationships 


NA 


NA 


2.86 


0.56 


2.89 


0.54 


Clear, high, and consistent behavioral 


expectations 


NA 


NA 


2.67 


0.46 


2.91 


0.48 


Clear, high, and consistent academic 


expectations 


NA 


NA 


3.13 


0.42 


3.11 


0.46 


Pedagogy: making connections and extensions 


NA 


NA 


2.74 


0.59 


2.82 


0.6 


Pedagogy: active learning 


NA 


NA 


1.82 


0.54 


1.88 


0.56 


Pedagogy: student-to-student interactions 


NA 


NA 


2.61 


0.61 


2.62 


0.6 



SOURCE: MDRC calculations from individual student school records from a large, urban school district (N=449). 



NOTES: The student achievement measures are Stanford Achievement Test (SAT) scores expressed in terms of 
normal curve equivalents (NCEs). The measures of student attitudes and behaviors and of school context are scales 
whose value ranges from 1 to 4, with higher values representing a greater quantity of a particular construct. 

“Prior Reading and Math Achievement is measured in Year 1. All other achievement and survey measures 
were measured in Year 2 and Year 3. 



Mathematics Achievement 

Table 3 shows the results of the analyses that examined the relationships between stu- 
dent engagement and perceived academic competence and mathematics achievement across all 
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School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 3 

Influence of Student Attitudes and Behavior on Mathematics Achievement 



Analysis Variables 


Standardized 

Coefficient 


Statistical 

Significance 


Student attitudes and behavior 3 


Engagement 


0.052 


* 


Perceived academic competence 


0.175 


** 


Student background 


Prior reading achievement 


0.543 


*** 


Race/Ethnicity 


Black 


-0.233 


*** 


Hispanic 


-0.036 


NS 


White 


0.169 


** 


Gender (males) 


-0.021 


NS 


SES (free or reduced-price lunch) 


0.002 


NS 


Grade (10th) 


-0.079 


* 


Effects of prior achievement on student attitudes 
and behavior 


PA on student engagement 


0.064 


* 


PA on perceived academic competence 


0.264 


*** 



SOURCE: MDRC calculations from individual student school records from a large, urban school district 
(N=449). 



NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *= 10 percent. 
NS=Not Significant. 

“The correlation between school engagement and perceived academic competence in Year 2 is quite 
high, r = .582; therefore, this parameter was estimated in the model to improve model fit. 



students. The results suggest that higher levels of both engagement and perceived academic 
competence in Year 2 preceded higher levels of math achievement in Year 3, although the in- 
fluence of perceived academic competence was approximately three times larger than that of 
engagement. These findings suggest: (1) how competent a student feels in previous school years 
plays a primary role in predicting how well she or he will perform on subsequent math tests and 
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(2) how engaged a student is in behavior associated with school success (such as doing home- 
work and trying hard in school) plays a secondary but meaningful role in determining the level 
of mathematics achievement. 

How big were these influences? A useful way to think about this question is to compare 
the magnitude of the influence with the amount of change that would normally be expected over 
the course of a school year. Previous research has shown that on average, secondary students 
improve about one-quarter of a standard deviation in their reading or mathematics achievement 
over the year. 33 The standardized regression coefficient for engagement is .052, or approxi- 
mately 2 1 percent of the change in mathematics scores that would be expected from the begin- 
ning until the end of the year. In comparison, the standardized regression coefficient for per- 
ceived academic competence — .175 — represents nearly 70 percent of the change in math 
achievement that would normally be expected over the course of the year. 

Several characteristics in students’ backgrounds were also significantly related to math 
achievement. As might be expected, prior math achievement had the strongest relationship with 
subsequent math performance, with an expected change in math scores of more than 7 NCE 
points. African-American students scored significantly lower than Hispanic or white students, on 
average, while white students scored higher than either Hispanic students or African-American 
students. Finally, tenth-graders in Year 3 scored slightly lower than eleventh-graders in Year 3. 

Within the same model predicting mathematics achievement, it was also possible to es- 
timate the influence of prior achievement on subsequent engagement and perceived academic 
competence. Students with higher math scores in Year 1 reported feeling more engaged and 
more academically competent in Year 2. Although prior achievement had a significant influ- 
ence on both psychological outcomes, the influence was much stronger for perceived compe- 
tence than for engagement. It appears that higher achievement test scores validated students’ 
sense of themselves as able learners and had a weaker, but still statistically significant, influence 
on their commitment to doing a good job on their schoolwork. 

Reading Achievement 

Table 4 shows the results of the analyses that examined the relationships between en- 
gagement in school and perceived academic competence and reading achievement. The findings 
for reading were quite different from those for mathematics. While previous level of perceived 
academic competence was an important antecedent of achievement in both reading and math, 
engagement showed a quite different pattern. In contrast to the positive influence of a prior level 



33 Kane (2004). 
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School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 4 



Influence of Student Attitudes and Behavior on Reading Achievement 



Analysis Variables 


Standardized 

Coefficient 


Statistical 

Significance 


Student attitudes and behavior 3 


Engagement 


-0.067 


* 


Perceived academic competence 


0.167 


** 


Student background 


Prior reading achievement 


0.722 


*** 


Race/Ethnicity 


Black 


-0.032 


*** 


Hispanic 


-0.087 




White 


0.019 


** 


Gender (males) 


0.041 


NS 


SES (free or reduced-price lunch) 


0.004 


NS 


Grade (10th) 


0.154 


* 


Effects of prior achievement on student attitudes 
and behavior 


PA on student engagement 


0.063 


* 


PA on perceived academic competence 


0.244 


*** 



SOURCE: MDRC calculations from individual student school records from a large, urban school district 
(N=449). 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *= 10 percent. 
NS=Not Significant. 

“The correlation between school engagement and perceived academic competence in Year 2 is quite 
high, r = .572; therefore, this parameter was estimated in the model to improve model fit. 

of student engagement on mathematics achievement, lower levels of prior engagement appeared 
to be associated with greater future reading success. 

As noted above, students would normally be expected to register an increase in reading 
of .25 standard deviation from the beginning to the end of the school year. But in this analysis, a 
one standard deviation increase in prior engagement produced a reduction in future reading test 
scores. Unlike student engagement, perceived academic competence was positively related to 
higher levels of future reading achievement. Students who felt more competent in Year 2 were 
likely to have higher reading achievement in Year 3. 
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To follow up on the unexpected negative relationship of engagement in school with 
reading achievement, a model was estimated that omitted perceived academic competence. As 
expected, the influence of engagement became somewhat stronger (b = .090) and positive. 
From these results, it appears that perceived academic competence had a mediating influence on 
the relationship between student engagement and reading achievement. This suggests that once 
the positive relationship between perceived academic competence and reading was accounted 
for, students displaying higher levels of engagement were less likely to be successful in their 
reading performance. 

As an additional follow-up, interaction analyses were conducted to determine if the in- 
fluence of engagement in school on reading achievement was different for students with differ- 
ent levels of confidence in their ability to be successful in school. The results from these find- 
ings suggest that engagement had a relatively strong influence on reading achievement (b = 
.197) for students with moderate levels of perceived competence, and a relatively weak influ- 
ence on reading achievement for students with high and low levels of perceived competence (b 
= .031 and b = .033, respectively). 

Interaction analyses were again conducted to examine more fully the relationship be- 
tween perceived academic competence and reading achievement as a function of levels of en- 
gagement in school. The results of these analyses indicate that at low levels of engagement, the 
relationship between perceived competence and reading achievement was somewhat negative 
(b =-.067); at moderate levels of engagement, the relationship was relatively strong (b = .289); 
and at high levels of engagement, the relationship was positive (b = .091), but less strong than at 
moderate levels of engagement. 

As with mathematics achievement, within the model of reading achievement it was 
possible to estimate the influence of prior achievement on subsequent engagement and per- 
ceived academic competence. Students with higher reading scores in Year 1 reported feeling 
more academically competent in Year 2; they also reported feeling more engaged in school, 
although this influence was weaker. 

Research Question 2: The Influence of School Context on Student Attitudes and 
Behaviors 

Since student attitudes and behaviors — especially perceived academic competence — 
appear to be important antecedents of math and reading performance, it is important to determine 
which conditions in classrooms support the development of higher levels of perceived academic 
competence and engagement in school. Key findings from analyses of this question include: 

• Finding 4. Several aspects of school context — teacher support, clear and 
consistent expectations of behavior, and student-to-student interactions in the 
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classroom — were significantly and positively related to engagement. 
Teacher support and expectations of conduct had an immediate influence on 
student engagement that was stronger than the longer-term influence, al- 
though both influences were statistically significant. 

• Finding 5. Teacher support and expectations of conduct were also signifi- 
cantly related to levels of perceived academic competence; again, the immedi- 
ate influence of these variables was stronger than their longer-term influence. 

• Finding 6. High academic expectations and a high level of student-to-student 
interactions in classroom instruction (such as student-led discussions) appear 
to have had a negative influence on perceived academic competence levels. 

• Finding 7. Active learning and making connections and extensions did not 
appear to be related to either engagement or perceived academic competence. 
Academic expectations also were not related to engagement. 

Tables 5 and 6 show the influence of characteristics of school context (that is, suppor- 
tive teacher relationships; clear, high, and consistent expectations; and high-quality pedagogy) 
on engagement in school and perceived academic competence, respectively. For each dependent 
variable, results for both the concurrent and the lagged models are displayed. 34 

Student Engagement 

Table 5 shows the results of the examination of the relationships between characteristics 
of school context and engagement in school. Both the concurrent and the lagged analysis mod- 
els show that students who reported experiencing higher levels of support from their teachers 
and greater understanding of the conduct expected of them also reported higher levels of en- 
gagement in school. The influence of these two variables in school context was stronger when 
students experienced these conditions during the same year that they rated their degree of en- 
gagement. Unlike conduct expectations, high and clear academic expectations were not related 
to student engagement, either immediately or in the longer term. 

Three variables measured the quality of pedagogy: active learning strategies, an empha- 
sis on making connections between class work and life outside school, and instructional activi- 
ties involving student-to-student interactions. In the lagged model, only student-to-student inter- 
actions were associated with subsequent engagement. When pedagogy and student engagement 

34 Lagged models consist of predictors measured at Year 2 and outcome variables measured during Year 3, 
except for prior achievement which was measured in Year 1. Concurrent models consist of predictors and out- 
come variables measured during the same school year. Year 3, except for prior achievement and student back- 
ground characteristics (measured in Year 1) and prior level of school engagement (measured in Year 2). 
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School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 5 

Influence of School Context on Student Engagement 





Concurrent Model 


Lagged Model 


Analysis Variables 3 


Standardized 

Coefficient 


Statistical 

Significance 


Standardized 

Coefficient 


Statistical 

Significance 


School context 


Supportive teacher relationships 


0.235 


** 


0.065 


* 


Clear, high, and consistent expectations” 


Academic expectations 


0.036 


NS 


0.034 


NS 


Conduct expectations 


0.187 


** 


0.071 


* 


High-quality pedagogy and strategies” 


Active learning 


0.049 


NS 


0.021 


NS 


Making connections and extensions 


0.046 


NS 


0.023 


NS 


Student-to-student interactions 


-0.013 


NS 


0.074 


* 


Student background 


Prior level of student engagement 


0.434 


*** 


0.487 


*** 


Race/Ethnicity 


Black 


0.039 


NS 


0.044 


NS 


Hispanic 


-0.028 


NS 


-0.044 


NS 


White 


0.006 


NS 


0.009 


NS 


Gender (males) 


-0.126 


** 


-0.152 


** 


SES (free or reduced-price lunch) 


-0.153 


*** 


-0.089 


* 


Grade (10th) 

Prior achievement (Year 1) 


-0.015 


NS 


-0.002 

0.196 


NS 

*** 



SOURCE: MDRC calculations from individual student school records from a large, urban school district 
(N=449). 



NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *= 10 percent. 
NS=Not Significant. 

'The overall lagged model accounted for approximately 45 percent of the variation in school engagement 
levels, scores (R-squared = .448). The overall concurrent model accounted for approximately 50 percent of the 
variance in school engagement levels (R-squared = .495). 

b To specify a correct model, both expectations variables were allowed to co-vary among each other. The 
correlation between academic and behavioral expectations was r^^E = .501. 

c To specify a correct model, all three pedagogy variables were allowed to co-vary among each other. 
Correlations among pedagogy predictors are as follows: r AL * MC = .432, r AL , ss = .349, and r MC „ ss = .416. 
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were measured simultaneously, there was no relationship between the two constructs — sug- 
gesting that, for this sample at least, pedagogy played little role in explaining students’ levels of 
engagement. One plausible explanation is that students consistently reported relatively low lev- 
els of high-quality pedagogy, leaving very little variation in their experience to be explained. 

Student background also played an important role in explaining Year 3 engagement in 
school. The strongest predictor of current engagement in school was students' previous level of 
engagement. Students who were more engaged during one year were much more likely to be 
engaged the next. Prior achievement was also significantly related to Year 2 engagement; that 
is, students who had higher levels of academic achievement in the first year of the study were 
more likely to be engaged in school the next year. However, the influence of prior achievement 
on engagement appeared to be moderated by perceived academic competence. When level of 
perceived academic competence was taken into account, the influence of prior achievement on 
engagement was reduced by two-thirds. This suggests that some of the variation in students’ 
levels of engagement in Year 3 was explained by how competent students felt the previous year, 
which was itself a function of how academically successful the students were in Year 1. 

Perceived Academic Competence 

Table 6 shows the results of concurrent and lagged analyses that examined the relation- 
ships between characteristics of school context and perceived academic competence. The analy- 
ses revealed that teacher support and high, clear, and consistent expectations of conduct con- 
tributed to students’ belief that they could be successful. The influence was both immediate 
(when all constructs were measured in the same school year) and longer term (when support 
and conduct expectations were measured one year and academic competence the next year), 
although the immediate influence was much stronger than the longer-term influence. 

However, in contrast to the student engagement results, higher levels of academic ex- 
pectations and students’ experience of more student-to-student interactions in their classes one 
year were associated with lower levels of perceived academic competence that year. That is, 
students reported that they felt less competent academically the more they experienced high, 
clear, and consistent academic expectations, and the more they were engaged in classroom ac- 
tivities that involved students working together. It may be that the more clearly students came to 
understand what constitutes high-quality performance — from examples provided either by 
their teachers or their peers — the more critical they became of their own work. 

Background characteristics also played an important role in explaining how academi- 
cally competent the students felt. The strongest predictor of perceived academic competence in 
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School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 6 



Influence of School Context on Perceived Academic Competence 





Concurrent Model 


Lagged Model 


Analysis Variables 8 


Standardized 

Coefficient 


Statistical 

Significance 


Standardized 

Coefficient 


Statistical 

Significance 


School context 


Supportive teacher relationships 


0.265 


** 


0.067 


* 


Clear, high, and consistent expectations” 


Academic expectations 


-0.097 


* 


0.037 


NS 


Conduct expectations 


0.292 


** 


0.062 


* 


High-quality pedagogy and strategies” 


Active learning 


0.004 


NS 


0.027 


NS 


Making connections and extensions 


0.045 


NS 


-0.039 


NS 


Student-to-student interactions 


-0.081 


* 


0.047 


NS 


Student background 

Prior level of perceived academic 


competence 


0.519 


*** 


0.505 


*** 


Race/Ethnicity 


Black 


0.078 


* 


0.087 


* 


Hispanic 


-0.033 


NS 


-0.058 


NS 


White 


0.029 


NS 


0.03 


NS 


Gender (males) 


0.008 


NS 


-0.056 


NS 


SES (free or reduced-price lunch) 


0.023 


NS 


-0.046 


NS 


Grade (10th) 

Prior achievement (Year 1) 


0.037 


NS 


0.04 

0.298 


NS 

*** 



SOURCE: MDRC calculations from individual student school records from a large, urban school district 
(N=449). 



NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *= 10 percent. 

NS=Not Significant. 

“The overall lagged model accounted for approximately 35 percent of the variation in perceived academic 
competence levels (R-squared = .398). The overall concurrent model accounted for approximately 49 percent of 
the variance in perceived academic competence levels (R-squared = .488). 

b To specify a correct model, both expectations variables were allowed to co-vary among each other. The 
correlation between academic and behavioral expectations was r AE * BE = .497. 

c To specify a correct model, all three pedagogy variables were allowed to co-vary among each other. 
Correlations among pedagogy predictors are as follows: r AL * MC = .441, r AL * ss = .347, and r MC * ss = .402. 
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Year 3 was perceived academic competence the previous year, controlling for prior achieve- 
ment. 35 Finally, African-American students reported higher levels of perceived academic com- 
petence than students in other groups. 

Research Question 3: Directionality of the Relationship Between School 
Engagement and Perceived Academic Competence 

The longitudinal data used in this paper make it possible to consider the directionality of re- 
lationships among constructs in the model, answering the question, “Which came first?” In the pre- 
vious analyses of the influences of student attitudes and behavior on reading and math achievement, 
the correlations between the measures of school engagement and perceived academic competence 
were quite high (r = .582 and r = .584, respectively). Since both engagement and perceived compe- 
tence appear to be important antecedents of academic performance, it is useful to examine the rela- 
tionship between the two variables. The key finding to emerge from this analysis is: 

• Finding 8. The data suggest that perceived academic competence is more 
likely to precede engagement in school than vice versa. 

The cross-lagged correlation model, shown in Figure 4, was used to estimate the direc- 
tionality of the influence between student engagement and perceived academic competence. 
The analysis examined the influence of perceived competence measured in Year 2 on engage- 
ment measured in Year 3 (controlling for Year 2 engagement); it also estimated the influence of 
Year 2 engagement on Year 3 perceived competence (controlling for Year 2 perceived compe- 
tence). Table 7 makes it clear that the lagged influence of perceived academic competence on 
student engagement was more than twice as large as the lagged influence of student engagement 
on perceived academic competence, although both influences were statistically significant. This 
suggests that perceived academic competence may precede engagement — that is, students who 
see themselves as academically competent become more engaged over time. The model also 
supports the theory — but less strongly — that students who are engaged early become more 
confident of their ability to be academically successful. 

Research Question 4: Directionality of the Relationships Between the Psychological 
and School Context Measures 

Do students who perceive the school environment as more supportive subsequently ex- 
perience greater engagement and perceive themselves as more academically competent? Or do 
students who are more engaged and feel more competent then come to believe that their school 



35 As noted above, prior achievement was related in an important way to students’ feelings of perceived 
academic competence, and this, in Utm, led to better academic outcomes. 



26 




School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 7 



Cross-Lagged Influence of Student Engagement and Perceived Academic 

Competence 





Lagged Model 


Model 


Standardized 


Statistical 




Coefficient 


Significance 


Student engagement influence on perceived academic competence 


0.069 


* 


Perceived academic competence influence on student engagement 


0.159 


** 



SOURCE: MDRC calculations from individual student school records from a large, urban school district 
(N=449). 



NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *= 10 percent. 

environment is more supportive? The fourth research question examines the directionality of the 
relationships between student engagement and perceived academic competence and those 
school context characteristics that the preceding analyses found to be significantly associated 
with engagement and competence. (As a reminder, teacher support, expectations of conduct, 
and student-to-student interactions in the classroom were identified as important in developing 
student engagement; and teacher support and expectations of conduct contributed to students’ 
expectations of success in school.) 

Key findings answering this research question include: 

• Finding 9. Students who were more confident academically and more en- 
gaged in learning tended to report more supportive relationships with teach- 
ers. There was also a weaker but nonetheless statistically significant influ- 
ence in the opposite direction (that is, students who had more supportive rela- 
tionships with teachers were more confident academically and were more 
engaged in learning). 

• Finding 10, Students who believed that the rules of conduct in their school 

were clear and fairly administered were more likely to feel engaged and aca- 

demically successful the next year. 
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• Finding 11. Students who said that they participated in learning activities 
that involved working with their classmates were more engaged in school a 
year later. 

To examine the directionality of the influence of important variables in the school con- 
text on school engagement and perceived academic competence, cross-lagged models (like the 
one shown in Figure 5) were analyzed for each pair of school context and student attitudes and 
behavior variables. Table 8 presents the results. 

School Context, Student Attitudes and Behavior, and Academic Achievement 

Table 8 

Cross-Lagged Influence of Student Engagement, Perceived Academic Competence, 

and School Context 





Lagged Model 


School Context Variables 


Standardized 


Statistical 




Coefficient 


Significance 


Teacher support 






Student engagement influence on teacher support 


0.202 


** 


Teacher support influence on student engagement 


0.065 


* 


Perceived academic competence influence on teacher support 


0.104 


** 


Teacher support influence on perceived academic competence 


0.067 


* 


Conduct expectations 






Student engagement influence on conduct expectations 


0.034 


NS 


Conduct expectations influence on student engagement 


0.071 


* 


Perceived academic competence influence on conduct expectations 


0.002 


NS 


Conduct expectations influence on perceived academic competence 


0.062 


* 


Student-to-student interactions 






Student engagement influence on student-to-student interactions 


0.046 


NS 


Student-to-student interactions influence on student engagement 


0.074 


* 



SOURCE: MDRC calculations from individual student school records from a large, urban school district 
(N=449). 

NOTES: Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *= 10 percent. 
NS=Not Significant. 
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Teacher Support and Student Engagement and 
Perceived Academic Competence 

Table 8 shows that the influence of student engagement and perceived academic com- 
petence on teacher support were stronger than the influence of teacher support on student en- 
gagement and perceived academic competence — from one and a half to three times greater. 
These findings suggest that students who are more academically confident and engaged tend to 
have more supportive relationships with teachers, and that teachers appear to be more suppor- 
tive of students whom they perceive as more involved in the academic process. There was a less 
strong but statistically significant finding that students who experienced high levels of teacher 
support one year were somewhat more engaged the next year and also felt more confident of 
their ability to do well. 

Conduct Expectations and Student Engagement and 
Perceived Academic Competence 

Expectations of conduct appeared to have a stronger influence on students' reports of 
engagement and their feelings of academic competence and success than vice versa. Students 
who reported that in their school there were clear and consistent behavioral norms and expecta- 
tions for both students and teachers felt more academically confident and more engaged in 
school the next year. The opposite was not the case: Students who were more engaged and felt 
more competent in their studies did not necessarily see the school climate as fair. 

Student-to-Student Interactions and Student Engagement 

Participation in classroom activities in which students worked together and helped each 
other learn was significantly related to how engaged students were in school a year later. Prior 
level of engagement was not related to how much students experienced student-to-student inter- 
actions in the learning process. 



Study Limitations 

The findings suggest interesting correlations among student achievement, engagement 
in school, perceived competence, and school context and also provide some evidence about the 
direction and nature of the linkages among these constructs. However, it is also important to 
recognize that the study has several limitations associated with the sample of students in the 
analysis, the data collection methods and measures, and the overall modeling approach and 
specification of the model. 

The sample for the study was drawn from an existing data set of students with unique 
characteristics. The students in the sample were primarily low-income, relatively low-achieving, 
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primarily Hispanic high school students. The findings may not generalize to students with dif- 
ferent characteristics, such as those who come from more advantaged backgrounds, are higher- 
achieving, or are predominantly white or African-American. Equally important, the students in 
the sample had persisted in school from eighth grade to tenth grade (for the first cohort of stu- 
dents) or from ninth grade to eleventh grade (for the second cohort). Students who stay in 
school may be different systematically and may have perceptions that are different from those 
who have already dropped out of high school. 

The data used in the analyses were based primarily on students’ self-reports (with the 
exception of academic achievement data). They did not involve teachers' reports or direct class- 
room observations, only students’ perceptions of classroom conditions. Without additional data, 
it is difficult to determine to what extent teachers demonstrated high-quality pedagogy and 
clear, high, and consistent academic and expectations of conduct. 

There are other issues associated with the data. First, the linkages among school con- 
text, engagement in school, perceived academic competence, and achievement might have been 
much more clearly delineated if they had been related to specific learning tasks. In this study, 
students’ survey responses were directed toward their current mathematics or English class in 
general, rather than to specific learning tasks. 

Second, the measures of mathematics and English achievement were not specific to any 
one classroom context, but rather to the cumulative set of English or math classes taken by the 
student over the course of his or her school career, and therefore may have been quite insensi- 
tive to the real influence of engagement or perceived competence. More sensitive measures 
might be developed to link these constructs to a specific classroom context. 

Third, the operational definitions of the constructs were based on a specific theoretical 
framework developed by the First Things First reform and drawn from existing data used in its 
evaluation. While the findings were limited to the confines of the measures and operational 
definitions associated with those measures, alternative conceptions of student engagement or 
high-quality pedagogy are possible. For example, a model that included items measuring more 
cognitive aspects of engagement might yield findings that differ from those presented here, 
which were based on self-reported behavioral engagement in school. 

Finally, the approaches to data analyses used here cannot yield definitive conclusions. 
While the lagged models allow for testing hypotheses about directional influences, only with 
caution should the findings be interpreted as causal. It is possible for a condition to precede an 
outcome without causing the outcome. This notion of causality is even less tenable in the con- 
current models, where no time lag exists between the measures. 
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Moreover, the general model hypothesized for this study (Figure 1) is only one of many 
plausible alternative models that could be postulated to examine the relationships among school 
context, engagement in school, perceived academic competence, and student achievement. In 
this study, pedagogy and academic expectations did not predict engagement in school and per- 
ceived academic competence. This finding does not necessarily mean that pedagogy and aca- 
demic expectations have no influence on student attitudes and behavior. Rather, a different 
model may be needed to explain the relationship of these constructs to the measures of student 
attitudes, behavior, and performance in the study. For example, pedagogy may have a direct 
influence on achievement, which then may lead to increased engagement in school and per- 
ceived academic competence. Other constructs not measured here, such as a highly developed 
sense of classroom community, quality or immediacy of feedback, or level of recognition and 
praise may actually be the critical supports in the development of engagement in school and 
perceived academic competence. 



Implications for Research and Practice 

The findings of this study have important implications for understanding how children learn 
in the classroom. Engagement in school was a critical predictor of mathematics achievement for 
high school students, a finding that is consistent with much of the literature. In contrast to the exist- 
ing literature, this study also suggests that perceived academic competence may play an even more 
important role than engagement in shaping achievement outcomes. The influence of perceived aca- 
demic competence on both reading and mathematics achievement was between two and four times 
larger than that of engagement in school. Furthermore, perceived academic competence appears 
both to predict engagement and to mediate the influence of engagement on achievement. 

The process, it seems, hinges on students developing a sense of efficacy and confidence 
about their ability to do well in school. Once students are confident of their ability to succeed, 
they become more engaged and learn more. On the other hand, students are not likely to attempt 
educational tasks when they feel they cannot succeed. And they are not likely to feel that they 
can succeed unless they have previously experienced success, along with the support needed to 
achieve that success. 

These findings suggest that the earlier schools and teachers begin to build students’ con- 
fidence in their ability to do well, the better off students will be. Because students’ notions of 
their capacity to be successful are so important to engagement in school and learning, school 
contexts should be designed in ways that support feelings of success. This study found that two 
factors in the school context — supportive teachers and clear and high expectations about be- 
havior — were key to enhancing the development of both perceived competence and engage- 
ment. Teachers whom students see as supportive and who set clear rules and guidelines about 
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behavior help create an atmosphere in which students feel in control and confident about their 
ability to succeed in future educational endeavors. 

The study also makes the case that student engagement is enhanced by learning activi- 
ties that involve student-to-student interaction. Other research suggests that challenging and at- 
tainable academic goals, teaching strategies that emphasize student collaboration, and subject 
matter that is meaningful and connected to students’ experiences also contribute to their feelings 
of academic competence and engagement. Teacher training strategies should focus on assisting 
teachers to create collaborative, supportive environments with high but achievable standards. 

It is puzzling that the study discerned so few statistically significant relationships be- 
tween the measures of high-quality instruction and those of perceived academic competence 
and engagement in school. It may be that students did not accurately interpret what was happen- 
ing in their classes. It may also be that the First Things First program had not been in place long 
enough for students to experience meaningful increases in academic expectations or active and 
connected learning strategies. There may simply have been too little variation across students’ 
instructional experiences to clearly understand the relationships between teaching strategies and 
perceived academic competence and engagement in school. 

The relationships that did emerge between these constructs tended to be negative (ex- 
cept for collaborative student-to-student interactions). This supports the conclusion that clear 
and high academic expectations and active and connected learning strategies may not in and of 
themselves be sufficient to increase students’ sense of involvement in learning and their belief 
that they can do well. If students view academic standards as so high that they are unattainable, 
they are unlikely to feel that they can be successful, and will be less engaged in schoolwork. 
Similarly, if students are exposed to active and connected learning strategies without sufficient 
scaffolding and support, the strategies will have a detrimental influence on their perceived ca- 
pacity for success and their subsequent engagement in school and learning. 

The findings also suggest that the supportive influence of school context may be more 
immediate than long term. The relationships between supportive relationships with teachers and 
clear behavioral norms on one hand, and perceived academic competence on the other, were 
much stronger in the same year than they were across years. Treating students well in the short 
term, then, may lead them to feel more confident, become more engaged, and learn more. But a 
positive initial experience is not enough, as the influences fade from one year to the next. To be 
successful in the long term, students may need supportive teachers and high-quality instruction 
throughout their high school careers. At the same time, the findings also suggest that an inter- 
vention that emphasizes supportive relationships, high and clear expectations, and high-quality 
instruction can make a difference to students at any point in their educational careers, and that 
these factors in the school context will produce strong and immediate influences. It is never too 
late to create high-quality academic environments with positive benefits for students. 
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Appendix 

Survey Items Used to Create Student Attitudes 
and Behavior Scales and School Context Scales 




Construct 


Survey Items 


Student Attitudes and Behavior Scales 


Student 

Engagement 

Scale 


• S5: How important is it to you to do the best you can at school? 

• S8: 1 pay attention in class. 

• S39: 1 often come to class unprepared. 

• S44: 1 work very hard on my schoolwork. 

• S53: 1 don’t try very hard in school. 


Student- 

Perceived 

Academic 

Competence 

Scale 


• S 14: 1 can’t do well in school. 

• S 18: 1 can do well in school if I want to. 

• S 16: 1 don’t know what it takes to get good grades in school. 

• S22: I'm not very smart in school. 

• S29: 1 can’t work very hard in school. 

• S35: Trying hard is the best way for me to do well in school. 

• S41: 1 don’t know how to keep myself from getting bad grades. 

• S50: I'm pretty smart in school. 

• S56: 1 am unlucky in school. 




School Context Scales 


Clarity of Expectations 


Teacher 
Support Scale 


• S 12: My teachers aren't fair with me. 

• S 17: My teachers like the other kids in my class better than me. 

■ S24: My teachers don’t make clear what they expect of me in school. 

■ S28: My teachers interrupt me when I have something to say. 

■ S3 1: My teachers like to be with me. 

■ S42: My teachers are fair with me. 

■ S47: My teachers’ expectations for me are way off base. 

■ S57: My teachers care about how I do in school. 


Academic 

Expectations 

Scale 


• S73: Your teacher makes clear to you how major assignments you’re given 
will be evaluated and graded. 

• S74: Your teacher makes clear to you how your overall grade will be 
determined. 

• S75: Your teacher makes clear to you examples of high-quality work that 
will lead to high grades. 

• S76: Your teacher makes clear to you examples of poor-quality work that 
will lead to low grades. 

• S77: Your teacher makes clear to you that are expected to come to class 
prepared. 

• S78: Your teacher makes clear to you that the work is meant to challenge you. 

(continued) 
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Construct 


Survey Items 


Academic 


• S79: You are expected to do your very best work all of the time. 


Expectations 


• S80: You can do good work if you try hard and participate fully in 


Scale (continued) 


classroom activities. 

• S81: Your teacher makes clear to you how to figure out specific steps you 
can take to improve your performance. 

• S85: You understand exactly what your grade is based on. 

• S90: Everybody is expected to take paid in classroom activities. 

• S93: It’s easy to “tune out” and not take part in class. 

• S96: All students are expected to work hard during class. 

• S97: Students can get away with not participating in class as long as they 
don’t make trouble. 


Conduct 


• S82: Your teacher makes clear to you how you should behave when 


Expectations 


working with other students in pairs or small groups. 


Scale 


• S83: Your teacher makes clear to you what student conduct is unacceptable. 

• S84: Your teacher makes clear to you what will happen to students who 

misbehave. 

• S7: A lot of students never get recognized for the good work they do. 

• S 13: Students in my school are expected to treat all of the adults in this 
school with respect all of the time. 

• S 15: The rules in this school are very clear. 

• S20: Students here get positive recognition when they do well in school. 

• S21: Students in my school are expected to treat each other with respect all 

of the time. 

• S30: All students get a chance to be recognized for the good work they do. 

• S36: Students get away with a lot in this school. 

• S52: All adults in this school treat all students the same when it comes to 

following the rules. 


High-Quality Pedagogy 


Active 


• S62: Listen to the teacher lecture for more than half the class period 


Learning 


• S63: Work in small groups or pairs 


Strategies 


• S64: Work individually on exercises from workbooks, texts, or handouts for 


Scale 


more than half the period 

• S65: Participate in discussions that students lead 

• S66: Discuss and ask each other questions about the work you’re doing 

• S67: Choose your own topics or problems to study 

• S68: Write essays, themes, poetry, or stories (solve math problems at the 
board) 

• S69: Use “real-life” situations in the work you are doing in class 

• S70: Write up projects you’ve done in class 

(continued) 
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Construct 


Survey Items 


Active 

Learning 

Strategies 

Scale 

(continued) 


• S7 1 : Work on projects that last several class periods 

• S72: Work on projects you help design 


Student-to- 

Student 

Interactions 

Scale 


• S91: Students help each other learn. 

• S92: Students go over and discuss each other’s work. 

• S94: Individual students speak about their work in front of the class. 

• S95: When students present their work to the class, other students ask 
questions and give feedback. 

• S98: Students review what they’ve learned with one another. 


Making 
Connections 
and Extensions 
Scale 


• S58: Learning to find more than one way of approaching a problem or issue 

• S59: Learning about the various sources of information that can be used for 
completing assignments and projects 

• S60: Making connections between what goes on inside and outside of 
school 

• S61: Making connections between what’s covered in your English/Math 
class and what’s covered in other classes 
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About MDRC 



MDRC is a nonprofit, nonpartisan social policy research organization dedicated to learn- 
ing what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness 
of social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best 
known for mounting rigorous, large-scale, real-world tests of new and existing policies 
and programs. Its projects are a mix of demonstrations (field tests of promising new pro- 
gram approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff bring an unusual combination of research and organizational experience to 
their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to 
learn not just whether a program is effective but also how and why the program’s effects 
occur. In addition, it tries to place each project’ s findings in the broader context of related 
research — in order to build knowledge about what works across the social and education 
policy fields. MDRC’s findings, lessons, and best practices are proactively shared with a 
broad audience in the policy and practitioner community as well as with the general pub- 
lic and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of pol- 
icy areas and target populations. Once known primarily for evaluations of state welfare- 
to-work programs, today MDRC is also studying public school reforms, employment 
programs for ex-offenders and people with disabilities, and programs to help low-income 
students succeed in college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Child Development 

• Improving Public Education 

• Promoting Successful Transitions to Adulthood 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the 
United Kingdom, MDRC conducts its projects in partnership with national, state, and 
local governments, public school systems, community organizations, and numerous pri- 
vate philanthropies. 




